DOCUMENT RESUME 



ED 291 766 



TM Oil 051 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



Greene^ Jennifer C. 

Participatory Evaluation and the Evaluation of Social 
Programs: Lessons Learned from the Field. 
Apr 86 

31p.; Paper presented at the Annual Meeting of the 
American Educational Research Association (San 
Francisco, CA, April 16-20, 1986). 
Reports - Evaluative/Feasibility (142) — 
Speeches/Conference Papers (150) 

MF01/PC02 Plus Postage. 

*Participation ; Participative Decision Making; Policy 
Formation; *Program Evaluation; *Social Science 
Research 

*Participatory Evaluation; ^Stakeholder Evaluation 



IDENTIFIERS 
ABSTRACT 

Issues associated with involvement of social program 
participants in program evaluation are analyzed. Although there seems 
to be a broad consensus of the need for participatory evaluation, the 
proposed rationales and concomitant benefits to evaluation practice 
appear to be quite diverse. Key issues include the rationale for 
"stakeholder** participation, definition of the ** stakeholder ,** 
definition of meaningful participation, types of evaluation and 
evaluation settings most appropriate for participatory approaches, 
roles of the participatory evaluator, and costs of participatory 
evaluation. Each issue is analyzed, and prescriptive upshots of the 
analysis are outlined. Viewing the optimal scientific process as 
democratic, one realizes that participatory approaches to program 
evaluation can fill one important niche in the desired linkage 
between social science and social policy/program delivery. (TJH) 



;****** ********************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

************************************************ir********************** 




Participatory Evaluation and tha Evaluation of Social Programs: 

Lessons Laarned from the Field 



Jennifer C. Greene 
Cornell University 



•PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTNER (ERIC) " 



U S OEPARTMENT OF EDUCATION 

Office of Educational Retearch and improvement 

EOUCAUONAL RESOURCES tNFORMATIQN 
CENTER (ERIC) 

Y This document has been reproduced as 
n received from the person or organizatidt 
oriOinatir>g it 

r MinOr changes have been maoe to tmprove 
reproduction quality 



Po'nts of vnw or opin ^ , Suited inlhiS docu 
mem 60 o* necessar ly represent official 
OERl position or policy 



Paper presented for a Symposium on Participatory Evaluation at 
tne Annual Meeting of the Amer-'can Educational Research Association, 
san Francisco, 1986. 



Pftrtlglgatorv Evaluation and the Eva luation of Social Proaramat 

Lessons Learne d from the Field 

One of the central themes in the field of program evaluation 
today is the involvement of program stakeholders in the evaluation 
process. While not a new idea (e.g., see Weiss, 1983a) , stakeholder 
participation is currently being advocated from such diverse 
perspectives as respon-aive evaluation (Cuba & Lincoln, 1981), 
evaluation as negotiation (Cuba & Lincoln, 1985), utilization- 
focused evaluation (Patton, 1978), postpositivist ctxtical 
multiplism (Cook, 1985), stakeholder-based evaluation (Bryk, 
1983; Gold, 1981), and "program improvement" evaluation (Cronbach 
& Associates, 1980) . Yet, within this consensus on the need for 
stakeholder participation, there is considerable diversity of 
proposed rationales and concomitant benefits to evaluation practice. 
In addition, for most of these rationales, neither the theoretical 
nor the operational elements of stakeholder participation are 
vtry well developed or understood. Such understanding, however, 
is required for the advancement of participatory approaches as 
viable evaluation alternatives. it is toward this understanding 
that this paper is offered. More specifically, this paper shares 
lessons learned from the field about several key issues related 
to the theory and practice of participatory evaluation. These 
issues and lessons are svunmarlzed in Table 1. 
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The primary field context for this discussion is a multiyear 
research study investigating selected aspects of participatory 
evaluation. Using a case study methodology, two participatory 
evaluations are being conducted in cooperation with two small 
local human service agencies (a county day care council and a 
city youth bureau). The participatory approach being used in 
this study is grounded in a utilization rationale. That is, the 
goal of enhancing evaluation use has guided all theoretical 
assessments and operational decisions about stakeholder 
participation. For this reason, many of the field-based lessons 
shared in this discussion are relevant to utilization-oriented 
participatory evaluation, but may or may not extend to participatory 
approaches guided by other rationales. 

Lessons Learn «>H 

S?aiuat^i^^ mional? for '^i-,K ^Ti m <lo r nrt-icin.^ion ...^t -^^ 

Th9 i ggue . As noted, stakeholder participation in the 
evaluation process has been advocated for several different 
purposes, including (a) increased utilization of evaluation 
results, (b) empowerment of especially lower status stakeholders, 
ind (c) greater congruence between evaluation and derision making 
processes (Mark & Shetland, i985a; Weiss, 1983b). A fourth 
rationale, (d) greater scientific validity or credibility of 
evaluation results, is suggested by the participatory emphasis of 
proponents of both "naturalistic" evaluation (Cuba & Lincoln, 
1981, 1985) and "postpositivist" evaluation (Cook, 1985) . For the 
former, participation is required for meaningful phenomenological 
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understanding, while for the latter, multiplicity of participants 
is one means to triangulate on valid results. 

LssSfins. One of the lessons learned about the rationale for 
stakeholder participation in evaluation is that the theory and 
operacionalization of stakeholder participation differ for different 
rationales. Thus, relatedly, it can be difficult to engage 
multiple rationales within a single participatory evaluation. 
For example, the criteria used to define and select stakeholder 
participants include (a) for a utilization rationale, program 
knowledge and status as a potential evaluation us«r, (b) for an 
empowerment rationale, low hierarchical status, (c) for a decision 
making rationale, status and power as a decision maker, and (d) 
for a validii.y/credibility rationale, diversity of perspectives. 
While not necessarily conflicting, these alternative criteria are 
not necessarilj complementary either. As a second example, 
participation itself is conceptualized somewhat differently within 
the different rationales. Key to meaningful participation within 
a utilization rationale is stakeholder control over decisions 
about substance or content. within an empowerment rationale, 
stakeholder participation in decisions about process is equally 
important. Both of these examples are addressed more fully below 
as separate issues. 

A second lesson learned concerns the meaning of utilization 
within a utilization rationale. Because participatory approaches 
are distinguished more by their process than by their products, 
utilization considerations should encompass both process and 
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product. Further, following Leviton and Hughes (1981) and 
others, utilization should be conceptualized broadly to include 
instrumental (concrete, action-oriented), conceptual 

(educational), and symbolic (persuasive, political) evaluation 
uses. For example, with this conceptualization, stakeholders 
involved in our two participatory evaluation case studies reported 
a number of conceptual and symbolic "benefits" or uses of their 
participation in the evaluation design process. These included 
learning more about and generating additional publicity for the 
progrM (Greene, 1985). 

A final lesson learned about participatory rationales pertains 
to their viability, both theoretically and operationally. The 
utilization rationale has some extant empirical support, though 
more from studies of discrete factors related to evaluation use 
(e.g., Aikin, Daillak, & White, 1979/ Thompson & King, 1981) than 
from wholesale implementations of pa. cicipatory approaches (though 
seeBryk, 1983; Donmoyer, 1983; Patton, nd; Smoler, 1984). Results 
to date from our own participatory research project support this 
trend, but also underscore the considerable challenge of 
implementation. Regarding other rationales, allow me simply to 
share my less informed thoughts. First, for empowerment, I wonder 
if evaluation is the best or even an appropriate strategy for the 
political goal of empowerment (see also Mark & Shetland, 1985a) . 
second, doesn't the decision making rationale still presume a 
rational mcdel of decision making or still presume that information 
can and will be regularly used to inform decisions? And hasn't 
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this assumption been severely questionned of late (Weiss, 1983b)? 
Third, the largely methodological nature of the validity/credibility 
rationale is appropriately and well supported by existing 
methodological arguments. 

Wh9 i9 a stakeholder? And who decides tf>^9 ia a ^i- »v ^^r.^ ^^jo 

Th? laaug . Following the original architect of stakeholder- 
based evaluation (Gold, 1981, 1983), the stakeholder concept is 
commonly defined as people whose lives are affected by the program 
and people whose decisions can affect the future of the program. 
As argued by stake (1975, 1983) , the essence of this concept implies 
risk and investment in the evaluand and is integrally related to 
issues of fairness and justice. Stakeholders are not simply people 
with information needs and thus potential evaluation users, but 
rather people with a vested interest in the evaluand, an interest 
or stake that can be lost, improved, protected, or treated 
(un) fairly. with this same conceptualization, Cuba and Lincoln 
(1981) identify all of the following as possible stakeholders in 
any evaluation setting: (a) people involving in developing and 
using the evaluand (program developers, funders of development 
efforts, current and future adapters, current funders, agency 
administrators, program staff, agency Boards, evaluation sponsors) ; 
(b) beneficiaries of the evaluand (direct and indirect) ; and (c) 
groups suffering a disadvantage related to the evaluand (groups 
excluded from participation, groups experiencing negative effects, 
groups suffering po^^ical disadvantage, groups suffering from 
lost opportunities and/ or resources) . 
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Mark and Shetland (l985a) extend this conceptualization by 
discussing tks values inherent in selecting stakeholder participants 
for an evaluation and by linking contrasting values to different 
participatory rationales. For example, utilization and decision 
making rationales both call for high-power stakeholder participants, 
While an empowerment rationale intentionally seeks to involve 
low-power or low-status participants. 

Finally, in this context, Weiss (1983b) has questionned the 
participatory assumption that there are program stakeholders who 
want an evaluation and are willing to participate in the process. 
While her own analysis lends support to this assumption, it is 
nonetheless germane to the present discussion. 

Lfiaaona. The lessons learned from the field regarding the 
definition and selection of stakeholder are twofold. First, our 
research has suggested that an appropriate candidate for 
participation in a utilization-oriented participatory evaluation 
is (a) a legitimate program stakeholder (a la Gold, 1981), (b) 
who has enough program knowledge to contribute to the process in 
ways meaningful to him/her and to the process, and(c) who defines 
him/herself as a participant (Greene, 1985) . That is, as implied 
by the first criterion, we did not find a link between the 
appropriateness (feasibility, meaning fulness, usefulness) of 
stakeholders as participants and their own power and status vis-a- 
vis the ovaluand, as suggested by Mark and Shetland (1985a) . Our 
stakeholder participants spanned a spectrum of power, which had 
little relation to the appropriateness of their participation. 
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Rath«r, &B iapliad by the second two criteria, appropriate 
participant status was linked to sufficient program knowledge, 
e.g., "I really don't know that much about how XXX [program] 
operates... so, I didn't see myself as a major contributor," and 
to stakeholders' willingness to place priority on their 
participation. This willingness, in turn, covaried with the 
strength and importance of stakeholders' own stake in the 
evaluation, which was largely self-defined, e.g., "i feel that they 
[XXX program] have done me such a service, so I'd like to help 
then in whatever I can" (client stakeholder), compared to, "Here 
was a rather small part of my job and I was being asked a lot of 
detailed information on a hat that I wear only 10 to 15% of the 
time" (other-agency stakeholder) . it should be reemphasized that 
these criteria are offered only for a participatory approach with 
a utilization rationale. Particularly the second criterion 
concerning sufficient program knowledge might be rejected by an 
empowerment-based participatory approach which emphasizes the 
evaluator's educational role and responsibilities. Finally, back 
to a utilization rationale, the empirical lack of support for 
Mark and Shetland's suggested need for high-power stakeholders 
nay be related to or confounded by the kind of evaluation being 
conducted (Bryk & Raudenbush. 1983). This issue is also addressed 
separately below. 

The second lesson learned about identifying stakeholders for 
a participatory evaluation concerns their actual selection. For 
this lesson, however, much remains to be learned. In our 
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utilization-oriented participatory case studies, we relied on the 
in-house perspective of agency administrators and program staff 
to select relevant stakeholder groups (from a Guba-and-Lincoln-like 
set) and to identify individuals within these groups as 
participants, using representativeness and diversity of perspective 
as major criteria. This procedure worked well, in that both initial 
groups of stakehol-lers were highly diverse and includ d, for 
example, individuals who were dissatisfied with the program. 
Yet, before advocating use of this procedure, the following concerns 
should be addressed. 

1. Representativeness and diversity are selection criteria 
under the logic that multiple perspectives from relevant constituent 
groups will contribute to the identification of the most important, 
i.e., useful, evaluation priorities and will facilitate multiple 
evaluation uses. Of concern here are the values inherent in this 
logic (Mark and Shetland, 1985a) and possible weaknesses in the 

. argument . 

2. Should "sufficient program knowledge" be added to the 
selection criteria, thus inevitably narrowing the candidate pool, 
but also avoiding possible negative "costs" of participation to 
those lacking such knowledge? 

3. What problems are incurred when stakeholder selectors 
have strong biases? what kinds of biases are particularly 
problematic? in such instances, what alternative sele-:tion 
procedures can be used? 
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What is meanlnafi^ Pflft i7ir?^iTTl'' 

IHfi—iasaa. The developers of the various participatory 
evaluation approaches consistently define stakeholder participation 
as shared decision making, rather than just advising or providing 
input to evaluation decisions. Stakeholders are "collaborators 
in inquiry" (stake, 1983:18). This consensus notwithstanding, 
the operational challenges of a shared decision making process 
are considerable (Greene, 1985; Murray, 1983). m addition, Weiss 
(1983b) has questionned the fundamental assumption of at least 
utilization-oriented participatory evaluation that stakeholders 
can identify in advance what they would like to learn from an 
evaluation. Greene (1985) and Mark and Shetland (1985a) have 
questioned the implications for meaningful participation of 
stakeholder limitations in understanding the technical elements 
of evaluation. Mark and Shetland (1985a) have also suggasted links 
between the desired nature a.id extensiveness of stakeholder 
participation and the varying participatory rationales. 

Lessons. The lessons learned from our research project about 
meaningful participation under a utilization rationale include 
the following, summarized here briefly. First, the operational 
challenges remain considerable; so, participatory procedures remain 
a priority area for research and development. Second, stakeholders 
can identify in advance their information needs, and a variety of 
procedures exist for this purpose, including our own four-phase 
question-i.ientification procedure (Greene, 1985), multiattribute 
utility scaling (Edwards & Newman. 1982), structured 
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conceptualization (Trochim & Linton, in press) , and time-honored 
Delphi techniques. Moreover, the time and resources required by 
such procedures are not only worth it, but in fact essential to 
realizing the utilization promise of participatory evaluation. 
In other words, in the absence of meaningful stakeholder 
participation in the initial identification of evaluation questions, 
utilization potential is substantially decreased. 

Third, in the shared derision making model developed in our 
research project, the evaluator has responsibility for structuring 
and guiding the evaluation process and for conducting all of the 
technical work, including data collection, analysis, and maintenanca 
of technical quality. Stakeholders ara then responsible for 
identifying and directing th substantive or programmatic content 
of the evaluation. Responsibility hero does not preclude 
participation by others. For example, stakeholders could and did 
participate in data collection, and evaluators could and did 
contribute to substantive discussions. To date, the process and 
products of these case study participatory evaluations have received 
satisfactory marks when graded on both utilization and technical 
criteria. This suggests that this shared decision making model 
has a future. Finally, additional salient elements of this modal 
are (a) the importance of providing tangible evidence of 
individuals' con w.'ibut ions to decisions, (b) the particular 
effectiveness of group discussions for this purpose, and (c) the 
value of iteration, or the provision of multiple and cumulative 
opportunities for participation, to the effectiveness of the model. 
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£25-.^ whftt Hinds — of evaluation and evalua<-Sr>n settings a ri> 

Parfrlg4pai;9rv gpprcaghes most acpronr^ai-o ;qW. i^v^ty 

effective? — ^ 

-_.isSilfi. Several authors have suggested that 

participatory approaches are better suited to formative, process, 
or implementation evaluations than to goal-oriented or summative 
evaluations (e.g., Cohen, 1983; stake, 1983; Weiss, I983b) . This 
is because the information needs of many stakeholders are likely 
to revolve around practical, operational aspects of program 
implementation (Cohen, 1983). in addition, formative or process 
evaluation designs are often less rigid than summative designs 
and thus can more flexibly respond to changes or shifts in 
participants' information needs (Weiss, 1983b). 

Relatedly, the .Inherent pluralism of all participatory 
approaches has implications for appropriate evaluation settings. 
Pluralism is embedded in all participatory rationales and thus 
permeates many aspects of the evaluation process, from the 
generation of evaluation priorities to the interpretation and use 
of evaluation results. This departure from reliance on a single 
evaluator or evaluation team, or the democratization of the 
evaluation process, is one of the most salient characteristics of 
participatory approaches. This shift to pluralism, however, remains 
encumbered by several concerns about the degree to which a single 
study -an effectively address the dive. ;e and, at times, 
invevitably conflicting needs of multiple stakeholders. These 
concerns include problems of achieving consensus on evaluation 
priorities (Weiss, 1983a, 1983b); of invoking an arbitrator role 
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for evaluatora, making them responsible "for adjudicating among 
rival interest* (including their own)"l (Cohen, 1983:93); and of 
undermining technical quality and, concomitantly, fulfillment of 
identified evaluation purposes by trying to be all things to all 
people. For example, stake (1983:25) observed that in one of the 
initial large scale trials of the stakeholder approach, "[t]he 
attempt to be useful to many may in fact have prevented it from 
being useful to any." 

These concerns about pluralism could imply that participatory 
approaches are best suited to evaluation settings that are 
relatively free from conflict and contentiousness . This suggestion, 
however, could very well seriously restrict the applicability of 
participatory approaches, as well as render moot a given rationale, 
e.g., to surface the heretofore hidden and divergent views of 
lev-status stakeholders. More commonly, these concerns about 
pluralism have yielded a call for multiple studies, each addressing, 
for example, the concerns of different stakeholder groups (Cohen, 
1983; Weiss, 1983b). This call has also been echoed in otuer 
quarters, for different though complementary reasons (Campbell, 
1984; Cook 1985; Cronbach & Associates, 1980). 

LsasSLA. Regarding this issue of appropriate evaluation 
types and settings, the lessons learned from our field project 
confirm a couple of suggestions in th<s literature, specifically: 

1. the appropriateness of participatory approaches for process 
evaluations. In both case studies, program implementation questions 
were identified as top evaluation priorities, and in both, the 

M 



13 

flexibility of our process evaluation designs allowed for iteration 
and shifts in data collection and analysis plans. 

2. the appropriateness of participatory approaches in small, 
local evaluation settings that are relatively free from conflict 
and contentiousness. The agencies participating in our research 
project are both snail, local agencies that are decentralized, 
collegial, and participatory themselves. Further, few conflicts 
of even a minor nature surfaced during either evaluation process. 

Given this evident lack of contrasts, however, our findings 
do little to inform these and other related concerns, which 
therefore remain important topics for further study. 
What roles are required of a p ^ rticlnat,nrY evaluatoT? 

The issue . Gold (1983) has acknowledged that "the express 
purpose of the stakeholder approach [is] to change role 
relationships between evaluators and users" (p. 67). Elaborating 
on this premise, ot.iers have contended that, in addition to the 
traditional methodological/technical expert, the participatory 
evaluator's roles include teacher, learner, facilitator, and 
arbitrator or negotiator (Greene, 1984; Mark & Shetland, 1985a; 
Weiss, 1983a, 1983b; see also note 1). As teacher, the evaluator 
educates stakeholders about the potential and limits of 
evaluation and/or the advantages and disadvantages of alternative 
methods. As learner, the evaluator seeks to understand 
stakeholders' multiple perspectives and views. As facilitator, 
the evaluator facilitates and enhances the process of group decision 
making. And as arbitrator, the evaluator negotiates and 
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••orc:iestrat«[3] the involvement of diverse interest groups" (vreiss, 
1983a:10). In this context, Weiss (I983a) has wondered whether 
these aultip:a role expectations for participatory evaluators are 
unreasonably high. And Greene (1984) has singled out the skills 
reqi:ired for effective group facilitation as posing particular 
challenges for traditionally trained evaluators. 

LSSSana. The major lesson learned from our participatory 
evaluation case studies is that, indeed, the roles of a 
participatory evaluator are multiple and diverse, in these studies, 
the evaluators served as technical ek-perts, technical advisers, 
teachers, learners, and facilitators. The arbitrator role was 
not required, primarily due to the aforementioned lack of conflicts. 
The second lesson learned was that each role does requir j a 
distinct set of skills and, for this evaxuator, the facilitator 
role was the most difficult. That is, my technical skills and 
teaching experience provided good support for all roles except 
that of facilitator, where my lack of training was, at times, 
acutely obvious. Finally, because it may be unreasonable to 
expect a single evaluator to have all these requisite skills, not 
to mention the mediation skills required of a negotiator, 
participatory evaluators may need to consider a team approach. 
Wh^t are the costs of nayt icipator^^ evaluatiori ? 

lbs — issue . One ob''ious sue of costs incurred in all 
participatory approaches to evaluation relates to increased burden, 
especially the additional time required of both evaluators and 
stakeholders, but also the additional skills required of both and 
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the risk of divisivaness and conflict (Greene, 1985; Mark and 
Shetland, .985a; Murray, 1983; Weiss, 1983b). Secondly, concerns 
have surfaced about possible tensions or conflicts between 
responsiveness to participatory political/organizational demands 
on evaluation and maintenance of technical/methodological evaluation 
standards of quality (Bryk & Raudenbush, 1983; Cohen, 1983; Greene, 
1983, 1985; Murray, 1983; Patton; nd; Shapiro, 1983). These 
concerns have been raised particularly for utilization-oriented 
participatory approaches as conflicts between efforts to maximize 
use and efforts to maximize quality. However, possible conflicts 
between responsiveness and technical responsibility can be readily 
anticipated for empowerment and decision making participatory 
rationales as well. 

A third and related cost of participatory approaches, argued 
most strongly by Bryk and Raudenbush (1983), concerns threats to 
the knowledge production function of evaluation. This argument 
is posed within the context of evaluating demonstration programs. 
Bryk and Raudenbush contend that in this context the primary 
stakeholders are not program staff or decision makers, but rather 
the research community or those social scientists who are concerned 
about the problem and who contemplate alternative remedies. 
Moreover, "evaluators- social responsibility in assessing 
demonstration programs implies norms for their conauot that resemble 
the norms of social science" (p. 99) . Yet, the "exclusive emphasis 
on use of evaluation findings [in the stakeholder-based approach] 
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seens likely to conpronise the knowledge productivity of applied 
social science" (p. 105) . 

Responses to this argument are threefold. First, as noted 
previously, participatory approaches may not be well suited to 
evaluations of demonstration programs in which researchers are 
the designated primary audience. Rather, process or implementation 
evaluations intended for program staff and decision makers as 
primary audiences may be the best forum for participatory 
approaches. Second, while the research community clearly 
constitutes a legitimate stakeholder group, it was precisely the 
dominance of evaluations that responded primarily to researchers' 
information needs that prompted the emergence of participatory 
approaches. According to Weiss (1983a j , such evaluations were 
viewed as largely narrow, unrealistic, irrelevant, unfair, and 
unused. Finally, the contention that program knowledge or theory 
is the exclusive province of the research community is being 
increasingly challenged. One form of this challenge suggests 
that it is not scientific theory, but rather the "local theory" 
understood by program staff and administrators that most directly 
affects service delivery (e.g., Cohen, 1983; Trochim, 1985). so, 
particularly in the context of evaluation for program improvement 
purposes, generating, understanding, and enhancing this local 
theory is a viable and important function of evaluation. Even 
more pointedly, following Weiss (1983b:91), such conceptual or 
"enlightenment" uses of evaluation may be the most appropriate 
and realistic goal for participatory approaches to evaluation. 
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LfiSAfiDfl. The lessons leamaa about costs from our two 
participatory evaluations supported many of these themes in the 
literature, with respect to burden, stakeholders did report time- 
related costs, though not so much the time required of them 
individually as that required by the overall process. Relatedly, 
several stakeholders experienced the process as quite disjointed. 
For the evaluators, this approach also required additional time, 
as well as the additional skills noted previously. Regarding 
conflicts with technical quality, a final assessment here awaits 
the results of an external review of evaluation quality, currently 
in process. At this point, our preliminary assessment suggests 
that the particular shared decision making model used avoided 
possible conflicts between use-oriented responsiveness and 
maintenance of technical quality. While allocating to stakeholders 
control over substantive issues, this model gave to the evaluators 
control over the structure and direction of the process and thus 
latitude to safeguard technical quality. Finally, the suggested 
threats to the knowledge production function of evaluation are 
not really germane to our participatory case studies, given their 
specified internal program improvement purposes. However, in 
this context, both evaluations did serve, in part, to generate, 
describe, and assess local theory. The usefulness of these results 
is currently being tracked. 

Concluaionj^ 

Thoughtful and concerned applied social scientists want to 
contribute positively to effective and just social policy and 
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program delivery. We "seek to improve [policymakers' and program 
deliverers'] conceptualizations by widening and clarifying the 
range of .. .choices, pointing out complexities that may be 
overlooked, and providing new perspectives... Social scientists 
also seek to provide information that is useful in policymaking 
[and program delivery] by documenting problems, monitoring 
implementation, and evaluating. . .alternatives" (Mark & Shetland, 
1985b:366-367) . 

This paper has suggested that, with a view of the scientific 
process as more fitting a democratic one, participatory approaches 
to program evaluation can fill one important niche in this desired 
linkage between social science and social policy/program delivery. 
The lessons learned to date about participatory evaluation support 
its contributive potential to this knowledge. They also begin to 
define and circumscribe this potential by identifying relevant 
contextual parameters. That is, like any other evaluation approach, 
a participatory one is not a panacea. Finally, like all good 
teaching, the lessons learned about participatory evaluation most 
importantly include further questions that need answering. 
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Notes 



This research was supported by New York State Hatch Grant 
#325410. 

lln Cuba and Lincoln's (1985) "fourth generation" model of 
evaluation, the evaluator's primary role is as a negotiator of 
pluralistic perspectives and interpretations. That is, in this 
model, the arbitration/negotiation role for evaluators occupies 
center stage and is viewed as positive and constructive, not as a 
problem. 
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Table 1 

Issues and lessons Learned About 



Issue 



Itiat Is the ratiimale for stakeholder participation in 
piTogran evaluaticri? 

Sugrgested rationales: 

1. increased utilization of evaluation results 

2. esafXMement of low status stakeholders 

3. greater a»gruenoe with decision making prxsoess 

4 . greater scientific validity/credibility of results 



Who is a stakeholder? And who decidas who is a 
stakeholder? 

As people yJhoee lives are affected by the program and 
people vHnose decisions can affect the future of the 
program, stakeholders include: 

1. people involved in developing and using the evaluand 

2. beneficiaries of the evaluand 

3. gvojpB sufferirg a disadvantage related to the 
evaluand 

A related issue is the values associated with defining 
and selecting rtak^lder participants. 
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Participatory Evaluation 



Lessons 



* The theory and operationalization of stakeholder 
participation differ for different rationales. So 
engaging multiple rationales within a single 
evaluation can be pFctolematic. 

* Within a utilization rationale, uses should be 
considered broaoly to e^xxniJass instnmental, 
cxjnoeptual, and synbolic uses of both the evaluation 
process and the evaluation results. 

* JI?® "J^ization rationale has some enpirical arnport, 
though its operational challenges remain considerable. 
Support for other rationales is less well established. 



* For a utilizatjan-oriented participatory evaluation, 
an appropriate stakeholder participant is: 

1. a legitimate stakeholder (a la Gold, 1981) 

2. with sufficient prognm knowledge to contribute 
In ways meaningful to hinv/her and to the pzxxxss 

3. v*io defines hlai/herself as a participant 

* Within a utilization rationale, procedures for 
selecting stakdiolder participants require further 
clarification of such concerns as: 

1. Are r^resentativeness and diversity cippropriate 
and defensible selection criteria? 

2. Should "sufficient program knowledge" be elided to 
the selection criteria? 

3. What problems are Incurred when the selectors 
have strong biases? 



Table ] 



Issue 



Vtiat is naemingfui participation? 

Within the view of neaningfta staloeholder participation 
as shared decision Baking, identified issues Include 
the operational challenges of such a process, 
stakeholder difficulties in identifying their 
infonnation needs, and stakeholder limitations in 
understanding the technical elements of evaluation. 



For v*iat kinds of evaluation and evaluation settings are 
participatory approadies nost appropriate or most likely 
to be effective? 

Participatory approaches are possibly voce appropriate 
for formative or process evaluations than for sunnative 
or product evaluations. 

The inherent pluralism of participatory apprxaches 
raises issues about serving ntatiple aiis within a 
single stuc^, including problans of: 

1. achieving consensus on evaluation priorities 

2. invoking a arbitrator role for evaluators 

3. undermining technical quality 

4 . identifying appropriate settings for participatory 
evaluations anVor invoking the need to conduct 
multiple studies to respond to this pluralism. 
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L (continued) 



Lessons 



* Ihe operational challaiges of shared decision making 
remain considerable, so participatory procedures 
remain a priority area for research. 

* Stakeholders identify in advance their information 
needs, using one of several extant procedures. 
Moreover, moaningfta stakeholder participation in 
this <3piestion-identif ication phase is integral to a 
utilization-oriented participatory approach. 

* Within a utilization rationale, a tenable shared 
decision model allocates responsibililty for substance 
to stakeholders and responsibility for process and 
technical work to evciluators. 



* Utilization-oriented participatory approaches are 
appropriate for process evaluation and for local 
settings that are relatively conflict-free. 

* Further research is needed on this issue. 
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Table 1 (continued) 



ISSUQ 



«iat roles are required of a participatory evaluator? 
Suggested roles: 

1. teciuiical/teethodological expert 

2. teacher and learner 

3. facilitator 

4. arbitratar/negotiator 

A related issue is lAiether or not such nultiple role 
e)^)ectations are unreasonably hig^. 



Wiat are costs of participatory eveduation? 
Suggested costs: 

1. increased turden (time, skills, risk of conflict) 

2. conflicts with technical quality 

3. threats to the knowledge production function of 
evaluation, e.g. , for evaluation of demonstration 
programs 
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Lessons 



* A participatory evaluator's roles are nultiple and 
diverse. In a oonflict-frBe setting with a utilization 
rationale, they Include all of the suggested roles 
except arbitrator/negotiator. 

* Each role recpiines a distinct set of skills. Thus, 
participatocy afproaches may require evaluation teans 
vs. single evzduators. 



* In a (conflict-free) utilization-oriented particdpato^ 
evaluation, the major burdens are increased time, 
increased length of the process, and additional akUl 
recpilranents for both evaluators and stakeholders. 

* *i "*^^^i2ation rationale, an appropriate shared 
decision making modex can avoid potential conflicts 
between use and quality by allocating to the evaluator 
latitude to safeguard technical quality. 



31 



